{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Yield From"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the last video we saw when we had two nested generators that we had to use a nested loop in order to iterate through both iterators:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def matrix(n):\n",
    "    gen = ( (i * j for j in range(1, n+1))\n",
    "            for i in range(1, n+1)\n",
    "          )\n",
    "    return gen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "m = list(matrix(5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<generator object matrix.<locals>.<genexpr>.<genexpr> at 0x0000028236EC2BF8>,\n",
       " <generator object matrix.<locals>.<genexpr>.<genexpr> at 0x0000028236EC2C50>,\n",
       " <generator object matrix.<locals>.<genexpr>.<genexpr> at 0x0000028236EC2EB8>,\n",
       " <generator object matrix.<locals>.<genexpr>.<genexpr> at 0x0000028236EC2F10>,\n",
       " <generator object matrix.<locals>.<genexpr>.<genexpr> at 0x0000028236EC29E8>]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose we want an iterator to iterate over all the values of the matrix, element by element.\n",
    "\n",
    "We could write it this way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def matrix_iterator(n):\n",
    "    for row in matrix(n):\n",
    "        for item in row:\n",
    "            yield item"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All we have done here is create a generator (iterator) that can be used to iterate over the elements of a nested iterator.\n",
    "\n",
    "We can then use it this way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "2\n",
      "3\n",
      "2\n",
      "4\n",
      "6\n",
      "3\n",
      "6\n",
      "9\n"
     ]
    }
   ],
   "source": [
    "for i in matrix_iterator(3):\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But we can avoid using that nested for loop by using a special form of `yield`: `yield from`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def matrix_iterator(n):\n",
    "    for row in matrix(n):\n",
    "        yield from row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "2\n",
      "3\n",
      "2\n",
      "4\n",
      "6\n",
      "3\n",
      "6\n",
      "9\n"
     ]
    }
   ],
   "source": [
    "for i in matrix_iterator(3):\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see we obtain the same result.\n",
    "\n",
    "We can think of \n",
    "```\n",
    "yield from <iterator>\n",
    "```\n",
    "as a replacement for the code:\n",
    "```\n",
    "for i in <iterator>:\n",
    "    yield i\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll come back to `yield from` in more detail, because there's a **lot** more to it than just a simple replacement for that inner loop!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's an example where using `yield from` can be quite effective.\n",
    "\n",
    "In this example we need to read car brands from multiple files to get it as a single collection.\n",
    "\n",
    "We might do it this way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "brands = []\n",
    "\n",
    "with open('car-brands-1.txt') as f:\n",
    "    for brand in f:\n",
    "        brands.append(brand.strip('\\n'))\n",
    "        \n",
    "with open('car-brands-2.txt') as f:\n",
    "    for brand in f:\n",
    "        brands.append(brand.strip('\\n'))\n",
    "        \n",
    "with open('car-brands-3.txt') as f:\n",
    "    for brand in f:\n",
    "        brands.append(brand.strip('\\n'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Alfa Romeo, Aston Martin, Audi, Bentley, Benz, BMW, Bugatti, Cadillac, Chevrolet, Chrysler, Citroën, Corvette, DAF, Dacia, Daewoo, Daihatsu, Datsun, De Lorean, Dino, Dodge, Farboud, Ferrari, Fiat, Ford, Honda, Hummer, Hyundai, Jaguar, Jeep, KIA, Koenigsegg, Lada, Lamborghini, Lancia, Land Rover, Lexus, Ligier, Lincoln, Lotus, Martini, Maserati, Maybach, Mazda, McLaren, Mercedes-Benz, Mini, Mitsubishi, Nissan, Noble, Opel, Peugeot, Pontiac, Porsche, Renault, Rolls-Royce, Saab, Seat, Å koda, Smart, Spyker, Subaru, Suzuki, Toyota, Vauxhall, Volkswagen, Volvo, "
     ]
    }
   ],
   "source": [
    "for brand in brands:\n",
    "    print(brand, end=', ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But notice that we had to load up the entire data set in memory.\n",
    "\n",
    "As we have discussed before this is not very efficient.\n",
    "\n",
    "Instead we could use a generator approach as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def brands(*files):\n",
    "    for f_name in files:\n",
    "        with open(f_name) as f:\n",
    "            for line in f:\n",
    "                yield line.strip('\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Alfa Romeo, Aston Martin, Audi, Bentley, Benz, BMW, Bugatti, Cadillac, Chevrolet, Chrysler, Citroën, Corvette, DAF, Dacia, Daewoo, Daihatsu, Datsun, De Lorean, Dino, Dodge, Farboud, Ferrari, Fiat, Ford, Honda, Hummer, Hyundai, Jaguar, Jeep, KIA, Koenigsegg, Lada, Lamborghini, Lancia, Land Rover, Lexus, Ligier, Lincoln, Lotus, Martini, Maserati, Maybach, Mazda, McLaren, Mercedes-Benz, Mini, Mitsubishi, Nissan, Noble, Opel, Peugeot, Pontiac, Porsche, Renault, Rolls-Royce, Saab, Seat, Å koda, Smart, Spyker, Subaru, Suzuki, Toyota, Vauxhall, Volkswagen, Volvo, "
     ]
    }
   ],
   "source": [
    "files = 'car-brands-1.txt', 'car-brands-2.txt', 'car-brands-3.txt'\n",
    "for brand in brands(*files):\n",
    "    print(brand, end = ', ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can simplify our function by using `yield from`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def brands(*files):\n",
    "    for f_name in files:\n",
    "        with open(f_name) as f:\n",
    "            yield from f"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Alfa Romeo\n",
      ", Aston Martin\n",
      ", Audi\n",
      ", Bentley\n",
      ", Benz\n",
      ", BMW\n",
      ", Bugatti\n",
      ", Cadillac\n",
      ", Chevrolet\n",
      ", Chrysler\n",
      ", Citroën\n",
      ", Corvette\n",
      ", DAF\n",
      ", Dacia\n",
      ", Daewoo\n",
      ", Daihatsu\n",
      ", Datsun\n",
      ", De Lorean\n",
      ", Dino\n",
      ", Dodge, Farboud\n",
      ", Ferrari\n",
      ", Fiat\n",
      ", Ford\n",
      ", Honda\n",
      ", Hummer\n",
      ", Hyundai\n",
      ", Jaguar\n",
      ", Jeep\n",
      ", KIA\n",
      ", Koenigsegg\n",
      ", Lada\n",
      ", Lamborghini\n",
      ", Lancia\n",
      ", Land Rover\n",
      ", Lexus\n",
      ", Ligier\n",
      ", Lincoln\n",
      ", Lotus\n",
      ", Martini, Maserati\n",
      ", Maybach\n",
      ", Mazda\n",
      ", McLaren\n",
      ", Mercedes-Benz\n",
      ", Mini\n",
      ", Mitsubishi\n",
      ", Nissan\n",
      ", Noble\n",
      ", Opel\n",
      ", Peugeot\n",
      ", Pontiac\n",
      ", Porsche\n",
      ", Renault\n",
      ", Rolls-Royce\n",
      ", Saab\n",
      ", Seat\n",
      ", Å koda\n",
      ", Smart\n",
      ", Spyker\n",
      ", Subaru\n",
      ", Suzuki\n",
      ", Toyota\n",
      ", Vauxhall\n",
      ", Volkswagen\n",
      ", Volvo, "
     ]
    }
   ],
   "source": [
    "for brand in brands(*files):\n",
    "    print(brand, end=', ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we still have to clean up that trailing `\\n` character..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "So, we are going to create generators that can read each line of the file, and yield a clean result, and we'll `yield from` that generator:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def gen_clean_read(file):\n",
    "    with open(file) as f:\n",
    "        for line in f:\n",
    "            yield line.strip('\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, this generator function will clean each line of the file before yielding it. Let's try it with a single file and make sure it works:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Alfa Romeo, Aston Martin, Audi, Bentley, Benz, BMW, Bugatti, Cadillac, Chevrolet, Chrysler, Citroën, Corvette, DAF, Dacia, Daewoo, Daihatsu, Datsun, De Lorean, Dino, Dodge, "
     ]
    }
   ],
   "source": [
    "f1 = gen_clean_read('car-brands-1.txt')\n",
    "for line in f1:\n",
    "    print(line, end=', ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, that works. So now, we can proceed with our overarching generator function as before, except we'll `yield from` our generators, instead of directly from the file iterator:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "files = 'car-brands-1.txt', 'car-brands-2.txt', 'car-brands-3.txt'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def brands(*files):\n",
    "    for file in files:\n",
    "        yield from gen_clean_read(file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Alfa Romeo, Aston Martin, Audi, Bentley, Benz, BMW, Bugatti, Cadillac, Chevrolet, Chrysler, Citroën, Corvette, DAF, Dacia, Daewoo, Daihatsu, Datsun, De Lorean, Dino, Dodge, Farboud, Ferrari, Fiat, Ford, Honda, Hummer, Hyundai, Jaguar, Jeep, KIA, Koenigsegg, Lada, Lamborghini, Lancia, Land Rover, Lexus, Ligier, Lincoln, Lotus, Martini, Maserati, Maybach, Mazda, McLaren, Mercedes-Benz, Mini, Mitsubishi, Nissan, Noble, Opel, Peugeot, Pontiac, Porsche, Renault, Rolls-Royce, Saab, Seat, Å koda, Smart, Spyker, Subaru, Suzuki, Toyota, Vauxhall, Volkswagen, Volvo, "
     ]
    }
   ],
   "source": [
    "for brand in brands(*files):\n",
    "    print(brand, end=', ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I want to point out that in this particular instance, we are using `yield from` as a simple replacement for a `for` loop. We could equally well have written it this way:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using `yield from`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def brands(*files):\n",
    "    for file in files:\n",
    "        yield from gen_clean_read(file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Without using `yield from`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def brands(*files):\n",
    "    for file in files:\n",
    "        for line in gen_clean_read(file):\n",
    "            yield line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Alfa Romeo, Aston Martin, Audi, Bentley, Benz, BMW, Bugatti, Cadillac, Chevrolet, Chrysler, Citroën, Corvette, DAF, Dacia, Daewoo, Daihatsu, Datsun, De Lorean, Dino, Dodge, Farboud, Ferrari, Fiat, Ford, Honda, Hummer, Hyundai, Jaguar, Jeep, KIA, Koenigsegg, Lada, Lamborghini, Lancia, Land Rover, Lexus, Ligier, Lincoln, Lotus, Martini, Maserati, Maybach, Mazda, McLaren, Mercedes-Benz, Mini, Mitsubishi, Nissan, Noble, Opel, Peugeot, Pontiac, Porsche, Renault, Rolls-Royce, Saab, Seat, Å koda, Smart, Spyker, Subaru, Suzuki, Toyota, Vauxhall, Volkswagen, Volvo, "
     ]
    }
   ],
   "source": [
    "for brand in brands(*files):\n",
    "    print(brand, end=', ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll come back to `yield from` in a lot more detail later when we study coroutines - there's a whole lot more to `yield from` than a replacement for a simple loop!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
